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Abstract 

We report on both analytical and numerical results concerning stochas- 
tic Hopfield-like neural automata exhibiting the following (biologically in- 
spired) features: (1) Neurons and synapses evolve in time as in contact 
with respective baths at different temperatures. (2) The connectivity 
between neurons may be tuned from full connection to high random di- 
lution or to the case of networks with the small-world property and/or 
scale-free architecture. (3) There is synaptic kinetics simulating repeated 
scanning of the stored patterns. Though these features may apparently 
result in additional disorder, the model exhibits, for a wide range of pa- 
rameter values, an extraordinary computational performance, and some 
of the qualitative behaviors observed in natural systems. In particular, we 
illustrate here very efficient and robust associative memory, and jumping 
between pattern attractors. 

The model and its motivation 

The field of neural automata, or cellular automata that are biologically in- 
spired and aimed at solving open issues in neuroscience, may be said to initiate 
with the Hopfield model [2]. This illustrates — on very simple and, consequently, 
rather unrealistic grounds — the property of associative memory, a task for which 
natural systems remain far from unbeaten by digital computers. Besides many 
other related studies the Hopfield model has been generalized along three 
interesting lines. That is, it has been shown that one may enhance the com- 
putational abilities of a Hopfield-like system by allowing for dynamic — instead 
of quenched — synapses OEI) by implementing dynamics in the computer via 
parallel — instead of sequential — updating 'S', T , and by considering scale-free 
— instead of extremely connected — network topologies [SJIH]. In this paper we 
report on some recent efforts along these lines. We describe a stochastic neural 
automata scheme that allows for explicitly studying the influence of the model 
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details on the resulting computational performance. In particular, we evaluate 
the effects of disorder perturbing both (statically) the network architecture and 
(dynamically) the synaptic strengths. The importance of these two details on 
the functioning of natural neural systems has been recently recognized, e.g. in 
Refs.^m] respectively. We conclude that the computational performance may 
indeed be importantly enhanced by appropriately tuning these details. This is 
illustrated here with some specific cases; the range of validity of our main results 
is known to go beyond these examples, however. 

Consider a network with a binary neuron variable, Si = ±1, at each node 
i = 1, . . . , TV. Assume pair synapses of weight Wij connecting the neurons at 
i and j, and define the degree ki of node i as the number of links from i to 
any other node. A scale-free network ^21 has power-law distributed degrees, 
P {k) ^ fc~^, while P (k) ^ S {k — N) , for the fully connected case (each neuron 
links to any other one in the network), and k has mean value CN, < C < 1, 
for the randomly ^-diluted case. The synaptic links are known to serve to store 
information. In the standard Hopfield model, for instance, one assumes M 
patterns consisting of binary variables, = ±1, ^ = 1, ...,M, and the weights 
are then set according to some learning rule. A familiar instance is the Hebb's 
rule for which the synapses take values Wij oc ■ 

The system configuration at time t involves the sets S = {s;} and W = {wij } 
which together determine the value of some energy function. To be specific, con- 
sider this to he H {S, W; t) = — i ^ . ^j'^ wtj (t) SiSj where the second sum goes 
over all nodes connecting to i. That is, we are assuming that there is an energy 
function with the Ising structure at each time t, and that synapses vary with 
t. For simplicity, we further assume that Wij (t) — Wij [/i (i)] = crA^~^^f ^'■'^j'*-*-', 
where cr is a normalization constant. This amounts to set each connection at 
time t equal to the value obtained from the /i th pattern via a learning rule. 
And our choice for this (see what follows) is such that, after time averaging, 
it results in the Hebb's rule (i-e. we average over all patterns with the 
same height). A main fact is that, consistently with the empirical observation 
that memory is a global dynamic phenomenon, we are assuming that all local 
synapses are set at each time by a given pattern, which is chosen according to 
some dynamics to be determined next. 

One is mainly interested in m = {m^; fi = 1, . . . , M}, where = m'^(S) 
is the overlap between the current state S and pattern /i. We assume ITS] 
that (S, /i) evolves in discrete time according to a transition rate T that may be 
decomposed as: T^' [S' ^ S] x Ts [fi' fi] , Here, is taken as a product of 
N terms ^ [(3qAH {s^ Si — ±s^)] (as corresponds to the neuron configuration 
varying by parallel updating or Little dynamics), and Ts = ^ [PiAH{p,' —^ /i)] . 
The function \E' {X) is arbitrary, e.g., of the Glauber type ^H], and each transi- 
tion rate involves both an inverse temperature P of the bath inducing its own 
stochasticity and the corresponding change AH of the chosen energy function. 

The fact that the neuron activities and the synaptic intensities are stochas- 
tically driven by different parameters, and Pi^, respectively — which one 
may imagine as the temperatures of two different baths in contact with each 
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set of degrees of freedom — endows the neural automata with a varied and in- 
teresting behavior. This is related to the fact that the competition between 
baths impedes the system from reaching thermal equilibrium; in general, the 
asymptotic state is instead a nonequilibrium steady state, which is more realis- 
tic than an equilibrium one ^Hl- Though the baths competition is, to the best 
of our knowledge, a novel feature within the present context, our model has 
some close antecedents ^1^1^]; these, however, mainly concern sequential 
updating, which in general results in a less efficient mechanism Further- 
more, our model describes the changes in the two subsystems on the same time 
scale, which is an interesting general situation with well-defined limits. That is, 
/3i — > oo corresponds to the freezing of synapses 2^ , large f3i is for slow synaptic 
kinetics jEli and small /3i describes extreme synaptic activity |51[TH]. 

This neural automata is also simple and versatile enough to allow for an- 
alyzing different network architectures. With this aim, we (i) first generated 
a network starting with rjo nodes and adding ij < ijo links at consecutive time 
steps with the preferential attachment rule. This, resulting in t] {N — rjo) links 
for A'' nodes, will be called the scale-free (Barabasi- Albert) network (SFN) |12) . 
(ii) A fully connected network (FCN) with N nodes has N'^ links, which is not 
realistic for a neural system. Instead, (iii) a meaningful alternative to the SFN 
case is the (highly) diluted network (DN) obtained by randomly suppressing 
links in the FCN until only rj {N — 770) are left. In order to have a true SFN 
with the small-world property, i.e., most of the nodes exhibiting small connec- 
tivity and a few hubs having its connectivity comparable to the network size A'^, 
one needs to restrict to 77 << iV. 

Scale— free topology of synapses 

The small-world property, which implies that the average path length between 
any two nodes is very small compared to the network typical length, is a suitable 
feature for an efficient neural system. In fact, this property has been reported 
to hold in many natural systems ^lED], including growing cultured neurons 
|1U|. and it was shown that the SFN can store and retrieve a given number of 
patterns with a lower computer-memory cost that the FCN [Sj . We shall briefly 
illustrate here (see also Ref. [5]) that the scale-free architecture may indeed 
enhance both the network associative performance and its robustness against 
thermal noise perturbations. 

The graphs in fig.l show the dependence of the stationary overlap on the 
neuron temperature, T = Pq^, for cases SFN and DN with 77 = 3 and M — 1. 
Notice that the SFN with competing temperatures coincides for M = 1 with 
the standard, equilibrium Hopficld model (with quenched synapses). Fig.l il- 
lustrates that the SFN makes indeed a better job for retrieving of information 
than the comparable DN at sufficiently high temperature, which is the rele- 
vant case for practical purposes. As M is increased, the performance tends to 
deteriorate in both cases. However, one then observes an intriguing behavior 
of the SFN at finite temperature. That is, there is a definite tendency of the 
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hubs, namely, nodes with k > kg (M, /3o, /3i) , to concentrate most of the system 
activity concerning the retrieval process. 

In order to illustrate the above property, we computed the mean local overlap 
associated to a node of degree k, ■m{k). The resulting graphs in fig. 2 clearly 
depict that m (k) — > 1 as /c increases at any temperature (large fluctuations 
simply reflect that the number of hubs is small for the network size used). 
This indicates how the hubs, the more the higher its connectivity degree, tend 
to become robust references for the process of associative memory. The state 
of boundary nodes, on the contrary, shows a poor correlation with the relevant 
stored pattern. This is in agreement with a previous observation at zero (neuron) 
temperature [211 . The above results triggered our interest in networks in which 
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Figure 1: Stationary overlap m'^ versus the logarithm of the neuron temperature 
(arbitrary units), averaged over 100 histories for N — 1600 neurons, M = 1, and 
»7 = 770 = 3, for the scale-free network (squares) and for the diluted network 
(circles). This illustrates a better performance of the former in general for 
practical purposes. 

the connectivity degree is power-law distributed, P (k) ^ k^^ . An issue is the 
possible influence of the parameter 7 on the system performance. In fact, the 
number of highly connected nodes tends to sharply decrease as 7 is increased, 
and it ensues 7 < 2 as a convenient range. Further study of this will be reported 
elsewhere (H). 

Escaping from the attractor 

Time behavior is also intriguing as illustrated, for instance, by the FCN archi- 
tecture, which is more amenable to analysis. Fig. 3 depicts the various phases 
exhibited by the system as the parameters [3q and [3i are changed. There is a 
ferromagnetic (F) phase in which the system shows associative memory, and 
a paramagnetic (P) phase — the upper stripe — lacking this property. This is 
familiar from the Hopfield model. A first novelty is that no mixed states JS| oc- 
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cur, which is computationally convenient. In addition, the nonequilibrium phase 
diagram in fig. 3 depicts a region between the P and F phases that exhibits emer- 
gent dynamic behavior confirming (and extending) a result in Ref. jllj. That is, 
the system in this region has dynamic associative memory, namely, after a tran- 
sient time in which one of the stored patterns is recovered, the system jumps 
to one of the other possible attr actors, and keeps indefinitely doing so. The 
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Figure 2: The local overlap, as defined in the main text, versus the connectivity 
degree k for different T, increasing from top to bottom, for the scale-free network 
in fig. 1. This illustrates a tendency of the hubs to control the retrieval of 
information. 

nature of this jumping deserves a comment. Simulations uncover that there is 
some non-trivial structure of time correlations. This is revealed by monitoring 
the time Tjy^,^ the system stays in attractor v before escaping to another one, 
say C- Within the largest jumping region, 0(11), t^^ happens to be practically 
independent of both v and (. That is, the system then stays the same amount of 
time wandering in each attractor. However, the probability of jumping between 
patterns depends on the activities, and non-trivial time correlations develop 
as the neuron temperature is lowered, namely, in region 0(1), where one even 
observes that t^_q ^ r^^i,. The behavior in 0(1) suggests using our algorithm to 
code spatial-temporal information \22\ . 

Summing up, we have illustrated several aspects of the behavior of Hopfield- 
like neural automata. This consists of neurons and synapses that evolve on the 
same time scale but subject to different thermal noises. Furthermore, differ- 
ent network architectures have been considered. It follows that a power-law 
topology, which is known to characterize many natural, including neural sys- 
tems is advantageous compared to the corresponding diluted network. We also 
find definite evidence that hubs, i.e. the few most highly connected nodes in 
a scale-free architecture, play a fundamental role in making the retrieval of 
information more robust and efficient. These findings, whose validity in natu- 
ral systems is to be checked, suggest paths for a convenient design of artificial 
systems. We have also demonstrated that, for appropriate parameter values. 
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neural automata perform much more efficiently when one lets the subsystem 
of synapses to constantly and coherently visit all the stored patterns in a con- 
venient way. This is fully consistent with two main empirical observations, 
namely, that memory is a global dynamic phenomenon, and that oscillations 
are essential to cortex functions. The neural automata has also been shown to 
exhibit in some situations spatial-temporal attractors, which may be relevant 
to simple olfactory processing, for instance. We acknowledge financial support 
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Figure 3: Phase diagram for the FCN automata depicting three coexistence 
curves that define several phases; see the main text. 
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